Papers with multimodal model
Multimodal, Multilingual Grapheme-to-Phoneme Conversion for Low-Resource Languages (D19-61)
Copied to clipboard
| Challenge: | Grapheme-to-phoneme conversion (g2p) is a task of predicting the pronunciation of words from their orthographic representation. |
| Approach: | They propose to leverage audio data as an auxiliary modality in a multi-task training process to learn a more optimal grapheme representation. |
| Outcome: | The proposed model reduces phoneme error rate to 2.46% on in-domain test set compared to unimodal spelling- pronunciation model. |
AMPS: ASR with Multimodal Paraphrase Supervision (2025.naacl-short)
Copied to clipboard
| Challenge: | Spontaneous or conversational multilingual speech presents many challenges for state-of-the-art automatic speech recognition systems. |
| Approach: | They propose a technique that augments a multilingual multimodal ASR system with paraphrase-based supervision for improved conversational ASR in multiple languages. |
| Outcome: | The proposed technique reduces word error rates by up to 5% on a state-of-the-art multimodal model . |
A Benchmark for Audio Reasoning Capabilities of Multimodal Large Language Models (2026.eacl-long)
Copied to clipboard
Iwona Christop, Mateusz Czyżnikiewicz, Paweł Skórzewski, Łukasz Bondaruk, Jakub Kubiak, Marcin Lewandowski, Marek Kubis
| Challenge: | Existing benchmarks for testing audio modality of multimodal large language models focus on testing audio tasks in isolation. |
| Approach: | They propose a new benchmark to assess multimodal large language models' ability to combine audio tasks. |
| Outcome: | The proposed benchmarks show that multimodal models can solve problems that require reasoning over audio signals with satisfactory results. |
KM-BART: Knowledge Enhanced Multimodal BART for Visual Commonsense Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for visual and language understanding are not capable of multimodal reasoning. |
| Approach: | They propose a Transformer-based sequence-to-sequence model capable of reasoning about commonsense knowledge from multimodal inputs of images and texts. |
| Outcome: | The proposed model performs state-of-the-art on the Visual Commonsense Generation task. |
Analyzing Modality Robustness in Multimodal Sentiment Analysis (2022.naacl-main)
Copied to clipboard
| Challenge: | despite its importance, little attention has been paid to improving the robustness of multimodal models. |
| Approach: | They propose simple diagnostic checks for modality robustness in a trained multimodal model . they find MSA models highly sensitive to a single modality, which creates issues . |
| Outcome: | The proposed checks show that models are highly sensitive to a single modality, which creates issues in their robustness. |
Multimodal and Multi-view Models for Emotion Recognition (P19-1)
Copied to clipboard
| Challenge: | combining lexical and acoustic information results in more robust and accurate models . combining both modalities may be a bottleneck in a deployment pipeline due to computational complexity or privacy constraints . |
| Approach: | They propose to combine acoustic and lexical information to provide a deployable acustic model . they use multimodal models and two attention mechanisms to assess the benefits of lexicals . |
| Outcome: | The proposed model outperforms the state-of-the-art on the USC-IEMOCAP dataset . it significantly surpasses models that have been exclusively trained with acoustic features . |
CMU-MOSEAS: A Multimodal Language Dataset for Spanish, Portuguese, German and French (2020.emnlp-main)
Copied to clipboard
AmirAli Bagher Zadeh, Yansheng Cao, Simon Hessner, Paul Pu Liang, Soujanya Poria, Louis-Philippe Morency
| Challenge: | Existing datasets in multimodal language are limited and disproportionately affect native speakers of other languages . authors propose a large-scale dataset for Spanish, Portuguese, German and French . |
| Approach: | They propose a large-scale multimodal language dataset for Spanish, Portuguese, German and French. |
| Outcome: | The proposed dataset is the largest of its kind with 40,000 total labelled sentences . it covers a diverse set topics and speakers and carries supervision of 20 labels including sentiment, emotions, and attributes. |
Different Data, Different Modalities! Reinforced Data Splitting for Effective Multimodal Information Extraction from Social Media Posts (2022.coling-1)
Copied to clipboard
| Challenge: | Recent multimodal information extraction approaches overestimate the significance of images. |
| Approach: | They propose a general data splitting strategy to divide social media posts into two sets to achieve better performance under information extraction models of the corresponding modalities. |
| Outcome: | The proposed method outperforms existing models on two different multimodal information extraction tasks. |
Deciphering Implicit Hate: Evaluating Automated Detection Algorithms for Multimodal Hate (2021.findings-acl)
Copied to clipboard
| Challenge: | Imlicit hate content has unusual syntax, polysemic words, and fewer markers of prejudice, e.g., slurs . multimodal content is harder to detect than unimodal content, such as memes . |
| Approach: | They evaluate the role of semantic and multimodal context for detecting implicit and explicit hate . they find that all models perform better on content with full annotator agreement . |
| Outcome: | The proposed model outperforms other models on implicit and explicit hate detection tasks because of its lower propensity towards false positives. |
RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER (2020.coling-main)
Copied to clipboard
| Challenge: | Named entity recognition (MNER) for tweets is a key task of many applications. |
| Approach: | They propose a pre-trained multimodal named entity recognition model based on Relationship Inference and Visual Attention (RIVA) for tweets. |
| Outcome: | The proposed model improves on the multimodal named entity recognition (MNER) task on tweets with the aid of visual clues. |
Speaker Naming in Movies (N18-1)
Copied to clipboard
| Challenge: | Identifying speakers and their names in movies is a primary task for many video analysis problems, such as automatic subtitle labeling. |
| Approach: | They propose a model that leverages visual, textual, and acoustic modalities in an unified optimization framework for speaker naming in movies. |
| Outcome: | The proposed model outperforms baseline models on the MovieQA 2017 challenge for speaker naming in movies and TV shows on visual, textual, and acoustic modalities. |
Multimodal Recommendation Dialog with Subjective Preference: A New Challenge and Benchmark (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing multimodal task-oriented dialog data fails to demonstrate the diverse expressions of user subjective preferences and recommendation acts in the real-life shopping scenario. |
| Approach: | They propose a multimodal task-oriented dialog dataset with subjective preferences and recommendation acts that is well-annotated with sales experts. |
| Outcome: | The proposed model is powered by a state-of-the-art multimodal model for these tasks. |
MM-SHAP: A Performance-agnostic Metric for Measuring Multimodal Contributions in Vision and Language Models & Tasks (2023.acl-long)
Copied to clipboard
| Challenge: | Vision and language models exploit unrobust indicators in individual modalities instead of focusing on relevant information in each modality. |
| Approach: | They propose a performance-agnostic multimodality score based on Shapley values that quantifies in which proportions a multimodal model uses individual modalities. |
| Outcome: | The proposed model can quantify in which proportions a multimodal model uses individual modalities for different tasks and datasets. |
Beyond Additive Fusion: Learning Non-Additive Multimodal Interactions (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Multimodal fusion addresses the problem of analyzing spoken words in the multimodal context, including visual expressions and prosodic cues. |
| Approach: | They propose to use multimodal fusion to separate unimodal, bimodal, and trimodal interactions in a multimodal model. |
| Outcome: | The proposed model separates unimodal, bimodal, and trimodal interactions while not degrading predictive performance. |
Multilingual and Multimodal Topic Modelling with Pretrained Embeddings (2022.coling-1)
Copied to clipboard
| Challenge: | a novel neural topic model for comparable data maps texts from multiple languages and images into a shared topic space. |
| Approach: | They propose a novel multimodal multilingual neural topic model that maps texts from multiple languages and images into a shared topic space. |
| Outcome: | The proposed model outperforms a zero-shot topic model in predicting topic distributions for comparable multilingual data and performs as well on unaligned embeddings as it does on aligned embeds. |
WhyAct: Identifying Action Reasons in Lifestyle Vlogs (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing systems for action recognition rely on pattern memorization and do not understand the action. |
| Approach: | They propose a multimodal model that leverages visual and textual information to automatically infer the reasons corresponding to an action presented in the video. |
| Outcome: | The proposed model leverages visual and textual information to automatically infer the reasons corresponding to an action presented in the video. |
Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Using large vision-language models to understand cultural contexts is a critical area of research. |
| Approach: | They conduct a thorough evaluation of multimodal models at different scales, focusing on their alignment with cultural values. |
| Outcome: | The proposed models show that they exhibit sensitivity to cultural values but their performance is highly context-dependent. |
Financial Forecasting from Textual and Tabular Time Series (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models that combine multiple data sources and combine them to form accurate financial predictions are challenging to model without inductive biases. |
| Approach: | They propose to use numerical financial results, macroeconomic states, and long financial documents to model company earnings relative to analyst expectations. |
| Outcome: | The proposed model outperforms existing models in a simulated trading environment and demonstrates that each modality contains unique information. |
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | a surge of deep learning applications for video understanding have led to major advancements in video-related tasks. |
| Approach: | They propose a multimodal video-based conversation model that merges a video-adapted visual encoder with an LLM and a dataset that is easily scalable and robust to label noise. |
| Outcome: | The proposed model can understand and generate detailed conversations about videos. |
Synthetic Multimodal Question Generation (2024.findings-emnlp)
Copied to clipboard
Ian Wu, Sravan Jayanthi, Vijay Viswanathan, Simon Rosenberg, Sina Pakazad, Tongshuang Wu, Graham Neubig
| Challenge: | Multimodal Retrieval Augmented Generation (MMRAG) is a powerful approach to question-answering over multimodal documents. |
| Approach: | They propose a synthetic data generation framework that leverages interplay between a retriever, large language model and large multimodal model to generate question and answer pairs directly from multimodal documents. |
| Outcome: | The proposed framework generates question and answer pairs from 1024 questions over Wikipedia documents and evaluates state-of-the-art models using it. |
How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models (2024.findings-emnlp)
Copied to clipboard
Jaeyoung Lee, Ximing Lu, Jack Hessel, Faeze Brahman, Youngjae Yu, Yonatan Bisk, Yejin Choi, Saadia Gabriel
| Challenge: | a growing influx of misinformation across news and social media is hampered by outdated foundation model training data. |
| Approach: | They propose to use large language models to scale up online policing mechanisms . they evaluate foundation model performance without continual updating . |
| Outcome: | The proposed model can improve performance without continual updating . the proposed model improves on two widely used benchmarks . |
“Mm, Wat?” Detecting Other-initiated Repair Requests in Dialogue (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current conversational agents (CAs) do not recognize repair initiation, leading to breakdowns or disengagement. |
| Approach: | They propose a multimodal model to automatically detect repair initiation in Dutch dialogues by integrating linguistic and prosodic features grounded in Conversation Analysis. |
| Outcome: | The proposed model integrates linguistic and prosodic features grounded in Conversation Analysis to detect repair initiation in Dutch dialogues. |
CLIPErase: Efficient Unlearning of Visual-Textual Associations in CLIP (2025.acl-long)
Copied to clipboard
| Challenge: | MU has gained significant attention as a means to remove the influence of specific data from a trained model without requiring full retraining. |
| Approach: | They propose a novel approach that disentangles and selectively forgets both visual and textual associations, ensuring that unlearning does not compromise model performance. |
| Outcome: | Experiments on CIFAR-100, Flickr30K, and Conceptual 12M show that CLIPErase effectively removes designated associations from multimodal samples in downstream tasks while preserving model performance on retain set. |